…ch point

Phase 3 (A-1223), infra + runner. The CI workflow that fans out the 1/5/10
sweep is intentionally not included yet (concurrency model is an open decision).

- bench-inclusion-sweep.env: bench-10tps's custom scrape pipeline + the
  tps-scenario topology trimmed to 150 full nodes (12x4 validators, 10 RPC),
  local eth-devnet, fake (fixed-delay) proving so proving never bottlenecks
  inclusion, P2P instrumentation on. Target TPS / sweep id are passed per point.
- full-node-resources-spot.yaml: full-node profile that *requires* spot
  (nodeSelector pool:spot + required gke-spot affinity + toleration), copying
  the prover-resources-dev.yaml pattern. Validators/RPC/prover stay on-demand.
- spartan/bootstrap.sh bench_inclusion_point: one sweep point at TARGET_TPS in
  its own namespace, all high-value load, scraped with a shared BENCH_SWEEP_ID
  (schema v4) via the existing bench_scrape path; reuses network_bench_upload.

Usable for local runs now: TARGET_TPS=5 BENCH_SWEEP_ID=incl-<date> \
  spartan/bootstrap.sh bench_inclusion_point bench-inclusion-sweep

Phase 3 (A-1223) CI wiring for the Set A inclusion sweep.

- nightly-bench-inclusion-sweep.yml: runs the three points SEQUENTIALLY (one
  ~150-full-node network at a time to bound cluster load), each in its own
  namespace (bench-incl-sweep-{1,5,10}tps), all tagged with a shared sweepId
  computed once in select-image. Each point chains deploy -> wait -> bench ->
  teardown; the next point's deploy waits on the previous point's teardown, so
  peak load stays at one network. A failed point drops only itself.
- ci.sh network-inclusion-sweep: threads TARGET_TPS / BENCH_SWEEP_ID /
  BENCH_SWEEP_LABEL into the EC2 bootstrap command (mirrors network-bench-10tps).
- bootstrap.sh ci-network-inclusion-sweep: runs one point via
  spartan/bootstrap.sh bench_inclusion_point; v4 run JSON (tagged sweepId) is
  uploaded to GCS inside that function. SKIP_NETWORK_DEPLOY=1 (deploy/teardown
  done by the workflow), mirroring the ci-network-bench-10tps path.

Run #95's proving sweep stalled with the agent fleet parked around ~50
replicas: under fake-realistic delays (PUBLIC_VM ~180s/job) the ~50 agents
could not drain a 32-checkpoint epoch's jobs within the proof-submission
window, so the proven tip never advanced past epoch 1 and the run ended in a
proof-window-expiry reorg. The KEDA bands only requested the full 200 agents
once the unassigned queue exceeded 200; at moderate backlog the fleet was
capped at 10. Ramp to 75 agents at queue>25 and the full 200 at queue>100 so
a fresh epoch's job burst pulls in the fleet it needs.

Tuning hypothesis to validate on the next run; node-pool/spot autoscaling may
still gate how quickly 200 agents actually schedule.

(cherry picked from commit ef43cb7)

… series

Verified against a live smoke run: the v4 scraper's memory and duration-histogram
series came back empty because the OTel->Prometheus exporter appends unit
suffixes the queries omitted:
- nodejs heap gauge is nodejs_memory_v8_heap_usage_bytes (was ..._usage)
- duration histograms carry a _milliseconds infix, e.g.
  aztec_proving_queue_job_duration_milliseconds_bucket,
  aztec_public_processor_tx_duration_milliseconds_bucket,
  aztec_prover_node_{block,checkpoint}_processing_duration_milliseconds_bucket

With these, memory (all roles), hint-gen, and proving-queue job_duration series
now populate. ELU + CPU were already correct. Full-node saturation remains
empty by design (FULL_NODE_INCLUDE_METRICS restricts full nodes to gossip).

…test fails

bench_inclusion_point ran the scrape only if the load test succeeded — a
non-zero test exit aborted the function first, discarding the inclusion records
and per-role/proving metrics it had already captured. A degraded point (e.g. a
higher-TPS point that misses its target, or any failed assertion) is exactly
where that data is most useful. Capture the test exit code, always scrape +
upload, then re-surface the failure via the return code so job status still
reflects it.

Two robustness changes to n_tps.test.ts, surfaced by a smoke run where the
inclusion benchmark mined 300/300 txs but still exited code 1:

- Replace the strict "successCount === totalHighValueSent" assertion with
  metrics.recordInclusionOutcome(mined, failed). A degraded point (a TPS target
  the network can't fully include) should report its mined-vs-failed counts +
  success_ratio, not fail the run — the ratio is itself the headline result we
  track. A total submission failure (nothing sent) is still a hard error.
- Wrap the afterAll wallet-cleanup loop and uninstallChaosMesh in try/catch so a
  teardown error (dead PXE, missing chaos-mesh CRDs on a run that didn't install
  it) is logged, not thrown — it was turning successful runs red.

tx_metrics.ts: TxInclusionMetrics gains recordInclusionOutcome, emitting
inclusion/mined_count, inclusion/failed_count, inclusion/success_ratio.

…value + low-value fill)

Each inclusion-sweep point now drives a fixed 1 TPS of high-value txs (the
measured inclusion lane) plus (TARGET_TPS - 1) TPS of low-value background
traffic, instead of all-high-value at the target. This matches bench_10tps and
measures what we actually care about: whether a properly-paying user's tx is
included promptly while the network runs at the target total TPS. The 10 TPS
point is now identical to bench-10tps (1 high + 9 low).

…eorgs

The high-value success/fail count came from waitForTx(waitForStatus: PROPOSED),
which throws/times out if a reorg drops a mined tx back to pending or the pool
evicts it — so a tx that WAS included got counted as failed under reorg churn
(inflating failed_count on reorg-heavy high-TPS runs). Inclusion is a
first-sighting event: the block-watcher already stamps minedAtMs idempotently
(never cleared on reorg). Add TxInclusionMetrics.wasMined(txHash) and, in the
waitForTx catch path, count the tx as included if it was ever observed in a
block — we don't care what happens to it afterwards.

Add ingressTxDurationP50/P99 from aztec.node.receive_tx.duration (histogram,
RPC-side tx-ingest cost on the submission path). This is the metric to watch
when RPC ingress is the bottleneck — it climbs as the RPC saturates, distinct
from the ingress-TPS rate. Flows into the existing timeSeries section (open map,
no schema change).

Reuse MEMPOOL_TX_MINED_DELAY but derive the value from the tx's persisted
receivedAt at the mined transition in handleMinedBlock, instead of the
add/remove timestamp map. The map-based recording fired on any removal
(eviction/deletion too), conflating mined delay with evicted delay; it was
also in-memory and lost across restarts.

handleMinedBlock now passes a per-tx minedDelayMs (now - receivedAt) through
onTxsMined, left undefined for txs hydrated from the DB on restart
(receivedAt === 0) so unknown receive times don't pollute the histogram. The
shared PoolInstrumentation map mechanism is retained for the attestation pool.

The pending-to-mined delay (aztec.mempool.tx_mined_delay) routinely exceeds the
60s ceiling of the shared ms-unit histogram view under load, saturating every
percentile at 60000. Add a name-scoped view with buckets from 1s to 10min,
placed before the generic ms view so it wins the bucket boundaries (the SDK
keeps the first-registered compatible storage; other ms histograms are
unaffected). Also fix the now-stale metric description.

…e event-loop delay

Three additive metrics in the v4 scrape output:
- Per-tx block build decomposition in summary: blockBuildPerTxMsAggregate,
  blockBuildMarginalMsPerTx (OLS slope), blockBuildFixedOverheadMsPerBlock
  (OLS intercept), blockBuildAvgTxsPerBlock — computed from per-block records.
- Pool-side pending-to-mined delay (aztec_mempool_tx_mined_delay) as both
  timeSeries (mempoolTxMinedDelayP50/P95/P99) and summary scalars, distinct
  from the client-observed inclusionLatency*.
- Event-loop delay (nodejs_eventloop_delay_mean) added to per-role saturation,
  surfacing the prover node's multi-second main-thread blockage at 10 TPS.

Validator and RPC pods previously requested only 0.5 CPU with a node-type=network
selector, so the scheduler bin-packed them across the shared cluster's mixed
2-/4-core network pools. Validators landing on 2-core nodes ran their 4 attesters
on too little CPU, fell behind on re-execution, and failed to attest ('Timed out
waiting for block with archive matching checkpoint proposal') — dragging the
committee below quorum and causing checkpoint prunes (reorgs) at 10 TPS.

Add bench resource profiles (request=limit, Guaranteed QoS) so every node gets
dedicated, un-contended cores and lands on an adequately-sized node:
- validator: 3.5 CPU / 12Gi  (one pod, 4 attesters, per dedicated 4-core node)
- rpc:       3.5 CPU / 8Gi   (dedicated 4-core; drops prod's preferred cores=2 affinity)
- full-node: 3.5 CPU / 8Gi   (kept on spot; packs onto 8-/32-core spot nodes)

Point both bench-inclusion-sweep envs at VALIDATOR/RPC/FULL_NODE_RESOURCE_PROFILE=bench.
Prover/broker keep dev-hi-tps (already explicitly sized, never bin-packed).

Adds a dedicated 4-core spot pool (spot_nodes_4core, t2d-standard-4) to the GKE
cluster. Benchmark full nodes request 3.5 CPU but there was no 4-core spot pool
(spot pools were 2/8/32-core), so they packed several onto 8-core spot boxes.
Pin the bench full-node profile to the new pool via a cores=4 nodeSelector so
each full node gets a dedicated 4-core spot node, without consuming the
on-demand pool that validators / RPC / prover run on.

Requires a terraform apply of the gke-cluster/cluster module before the next
bench deploy, otherwise the cores=4 + pool=spot full-node pods stay Pending.

…2s slots

The inclusion sweep (1/5/10 TPS) is a superset of the single-point 10 TPS
nightly (it reuses the bench-10tps load model), so retire nightly-bench-10tps.yml
and let nightly-bench-inclusion-sweep.yml be the nightly TPS/inclusion benchmark.

Rebase bench-inclusion-sweep.env on the smoke env (same 12x4 validator / 10 RPC /
10 full-node topology on dedicated guaranteed-QoS bench profiles, RPC ingress
off, fake proving) but at a representative 72s slot duration and 800 tx per
checkpoint. 8-slot epochs keep nightly committee warm-up short. Test duration is
already 10 min (bench_inclusion_point default TEST_DURATION_SECONDS=600; the
workflow does not override it).

Replace bench-inclusion-sweep.env with the smoke env's topology (12x4 validators,
10 RPC, 10 full nodes on dedicated guaranteed-QoS bench profiles, RPC ingress off,
fake proving, 8-slot epochs) but at a representative 72s slot duration and 800 tx
per checkpoint. Drops the prior production timing / RPC LoadBalancer / per-block
cap that the sweep no longer uses.

…l/bench-integration-v5

# Conflicts:
#	.github/workflows/nightly-bench-inclusion-sweep.yml

…etwork

Previously each TPS point deployed and tore down its own network (3x deploy +
teardown). Switch to deploying a single network, running the 1/5/10 points
sequentially against it (the scraper drains the mempool to zero between points),
and tearing it down once. Cheaper, faster, and matches how it's run locally.

deploy/wait/teardown move to the top-level workflow (run once); the reusable
bench-inclusion-point.yml is now just the bench step (one point, against the
existing network, SKIP_NETWORK_DEPLOY=1), called 3x with needs chaining so the
points don't overlap. Later points are gated on `wait` success so a failed
point still drops only itself.

Capture each pod's aztec-container CPU/memory requests and limits during
the infrastructure scrape, exposed as `resources` per node and a deduped
`resourceProfiles` per role, so a run records the machine sizing the
validators/rpc/full-node/prover pods actually got (request==limit for the
guaranteed-QoS bench profiles).

Also anchor the role-classification patterns to the namespace-stripped pod
name so the L1 infra pods (eth-validator / eth-execution / eth-beacon) are
no longer misread as aztec roles — an unanchored /-validator/ matched
"<ns>-eth-validator-0" and polluted the validator role.

The node emits aztec.validator.attestation_failed_node_issue_count (the
attester-side timeout/missing-data counter that diagnosed the 10 TPS reorgs)
alongside the bad-proposal and success counters, but the bench scraper never
queried them — only the proposer-side collect-duration gauges reached the JSON.

Add three timeSeries (node-issue and bad-proposal failures broken down by
error_type, plus a success rate for a failure-ratio denominator) and three
summary scalars totalling each outcome over the observed window, so the
attestation node-issue count — the signal upstream of committee quorum loss —
is recorded per run.

prune_count was incremented only by the proven/epoch prune (handleEpochPrune),
so the pending-chain reorgs that drive world-state's "Chain pruned to block N"
— proposed blocks dropped because their slot ended without a checkpoint, because
they conflicted with an L1 checkpoint, or because their matching proposed
checkpoint never arrived before the deadline (pruneOrphanBlocks) — incremented
no metric and were observable only in logs.

Add a prune_type attribute (unproven / uncheckpointed / l1_conflict / orphan) to
prune_count and tag all four emit sites. Scraped per pod, this distinguishes a
single straggler node's local re-sync from a network-wide reorg.

…breakdown

reorgCount counted `Chain pruned` log events network-wide, which conflates one
straggler node repeatedly re-syncing its local view with a genuine network
reorg. The 10 TPS sweep showed this: 3 "reorgs" that were all one saturated
validator pruning its own view three times while the canonical chain never
moved.

Re-point reorgCount to count DISTINCT PODS that experienced a pending-chain
prune (archiver prune_count, prune_type in uncheckpointed/l1_conflict/orphan),
so a straggler reads as 1 and a network reorg as many. Add a prunesByType
summary breakdown and a pruneCountByType time series. deepestReorgBlocks stays
log-derived (no depth metric). Requires the prune_type-attributed metric (new
image); reads 0 on older images, cross-checked by the chainPruned events.

Add a throughputTps series carrying three labeled sub-series (source =
gossip | rpc_receive | mined) so the dashboard overlays tx rate at three stages
on one chart: gossip propagation (avg per-node accepted tx-topic gossipsub
messages — every node accepts each unique tx once, so avg approximates the
unique-tx propagation rate; sum would overcount by pod count), RPC ingress
(receiveTx on the submission node), and mined (archiver block tx rate, avg
across archivers). Query validated against retained Prometheus data.

Add p2pBandwidthBytesPerSec carrying four labeled series (recv/sent × avg/max
per pod) from the gossipsub RPC byte counters. Per pod rather than summed so the
hottest-pod (max) series surfaces gossip bandwidth that isn't evenly spread — at
10 TPS the hottest pod moved ~4 MB/s each way vs ~2.3 MB/s average. No
libp2p-level transport metric is exported, so the gossipsub byte counters are
the bandwidth signal. Query validated against retained Prometheus data.

…licate-tf-blocks

The spot_nodes_4core node pool definition belongs with the gke-cluster terraform
cleanup, not the bench branch. Removed here and applied to
stack/chore-remove-duplicate-tf-blocks. The pool is already applied to the live
aztec-gke-private cluster, and the bench full-node resource profile that targets
it (cores=4 selector) stays on this branch.

Strip internal tracker issue IDs and the 'Set A' project label from code and
workflow comments; the repo is public so these convey nothing to external
readers. Behaviour unchanged (comment-only edits).

Tag each chainPruned event with podCount (distinct pods that pruned to the same
block within the dedupe window). reorgCount now counts only genuine chain reorgs
(podCount >= 2 — the canonical chain rewound across nodes); single-node local
prunes (a straggler that diverged and re-synced) are counted separately as
nodeLocalPruneCount and no longer inflate reorgCount or deepestReorgBlocks. This
is log-derived, so it is accurate regardless of node image. Verified on the
1/5/10 TPS runs: 0 chain reorgs, with the 5/10 TPS prunes correctly classified
as node-local.

Refine reorg detection: tag each chainPruned event with validatorPruneCount
(distinct validator pods that pruned to that block) and compare against the
validator fleet size (distinct validators that logged l2-block-handled).
reorgCount now counts only events where ALL following validators rewound to the
same block — a true canonical reorg. A prune reported by a subset of validators
(or by non-validator followers) is a local divergence (nodeLocalPruneCount), not
a reorg. Verified on 1/5/10 TPS: validatorCount=12, 0 chain reorgs (prunes had
0-1 validators each).

Adds an optional run.benchmarkType enum (ingress-inclusion, simulated-proving,
real-proving, block-capacity) so the dashboard can group a day's sweeps by
benchmark kind. Bumps the run schema 4->5 and the index schema 1->2.

- bench_scrape.ts: --benchmark-type arg / BENCH_BENCHMARK_TYPE env, emitted in run
- bootstrap.sh: inclusion point passes ingress-inclusion; benchmarkType copied
  into each index.json entry; upload gate now expects schema "5"

…tstrap

The single-shot 10 TPS bench (bench_10tps / bench_10tps_cmds) is superseded by
the inclusion sweep (bench_inclusion_point, run at 1/5/10 TPS). Removes the dead
functions and the case-dispatch entry; the shared scraper at
scripts/bench_10tps/bench_scrape.ts is still used by the sweep and is untouched.